Objective¶
Using the data collected from existing customers, build a model that will help the marketing team identify potential customers who are relatively more likely to subscribe term deposit and thus increase their hit ratio.
Import Libraries¶
In [1732]:
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import BaggingClassifier
from sklearn.ensemble import AdaBoostClassifier
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import confusion_matrix, recall_score, precision_score, f1_score, roc_auc_score,accuracy_score,roc_curve
from sklearn.metrics import classification_report
from sklearn.preprocessing import LabelEncoder
from sklearn import preprocessing
from scipy import stats
from sklearn.tree import export_graphviz
from sklearn.externals.six import StringIO
from IPython.display import Image
import pydotplus
import graphviz
import warnings
warnings.filterwarnings('ignore')
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity='all'
Import Dataset¶
In [1733]:
# Read the CSV file
df=pd.read_csv('bank-full.csv')
Univariate analysis¶
Check datatype, Summary, shape, Null values¶
In [1734]:
df.info() # Info about dataframe, columns, data types, Null values
In [1735]:
print('Shape of the dataframe is:',df.shape) # Shape of data frame
print('-----First five rows are-----')
df.head() # First five rows
print('-----Last five rows are-----')
df.tail() # Last five rows
print('-----Number of unique values-----')
df.nunique() # Number of unique values in each column
print('-----Number of Nulls-----')
df.isnull().sum() # Check Null values
df.describe().transpose() # Statistical summary
Out[1735]:
Out[1735]:
Out[1735]:
Out[1735]:
Out[1735]:
- The independent variables are --> age, job, marital, education, default, balance, housing, loan, contact, day, month, duration, campaign, pdays, previous, poutcome
- The dependent variable is --> Target
- The categorical columns are represented as Object datatype and hence need to be converted.
- There are 45211 rows and 17 columns
- There are outliers especially for age, balance and campaign and data is right skewed
- Data for age and campaign seems to be real world situation
In [1736]:
# Convert to categorical data type
df['job']=df['job'].astype('category')
df['marital']=df['marital'].astype('category')
df['education']=df['education'].astype('category')
df['default']=df['default'].astype('category')
df['housing']=df['housing'].astype('category')
df['loan']=df['loan'].astype('category')
df['contact']=df['contact'].astype('category')
df['month']=df['month'].astype('category')
df['poutcome']=df['poutcome'].astype('category')
df['Target']=df['Target'].astype('category')
# Check datatyoes
df.dtypes
Out[1736]:
Age¶
In [1737]:
plt.figure(figsize=(14,5))
sns.distplot(df['age']);
In [1738]:
df['age'].unique() # Check unique values
df['age'].isnull().sum() # Check null
df['age'].describe() # Check mean, min, max, std, quartiles
df['age'].median() # Median value
Out[1738]:
Out[1738]:
Out[1738]:
Out[1738]:
- Mean for Age is 40.936210
- Median (Q2) for Age is 39
- Not much difference between mean and median age
- Min value for Age is 18
- Max value for Age is 95
- Q1 for Age is 33
- Q2 for Age is 39
- Q3 for Age is 48
- As we can see from the plot above data is right skewed
In [1739]:
# Q1 and Q3 values from above
outliers_lower=33-1.5*(48-33)
outliers_upper=48+1.5*(48-33)
print(outliers_lower)
print(outliers_upper)
# Number of outliers
print('Number of outliers lower:',df[df['age']<outliers_lower]['age'].count()) # Lower
print('Number of outliers upper:',df[df['age']>outliers_upper]['age'].count()) # Upper
In [1740]:
plt.figure(figsize=(3,5))
sns.boxplot(x='age',data=df, orient='v');
Analysis for age¶
- The minimum age is 18 and maximum age is 95 so the spread is across
- There is little variance between mean and median values and we do not see any null values
- The data for age is right skewed as can be seen from distplot and boxplot above
- Though there are outliers but they seem to be real world situation
Job¶
In [1741]:
plt.figure(figsize=(20,10))
sns.countplot(df['job'],data=df)
plt.xlabel('Job', fontsize=15)
plt.ylabel('Count', fontsize=15)
plt.show();
In [1742]:
df['job'].unique() # Check unique values
df['job'].isnull().any() # Check null
Out[1742]:
Out[1742]:
In [1743]:
df['job'].value_counts()
df['job'].value_counts(normalize=True) # value counts %age
Out[1743]:
Out[1743]:
Analysis for job¶
- For 288 customers the job information is not known
- There are more customers in blue-collar (21.5%) and management (21%) jobs
- There are no null values
Marital¶
In [1744]:
plt.figure(figsize=(5,5))
sns.countplot(df['marital'],data=df)
plt.xlabel('Marital Status', fontsize=15)
plt.ylabel('Count', fontsize=15)
plt.show();
In [1745]:
df['marital'].unique() # Check unique values
df['marital'].isnull().any() # Check null
df['marital'].value_counts() # value counts
df['marital'].value_counts(normalize=True) # value counts %age
Out[1745]:
Out[1745]:
Out[1745]:
Out[1745]:
Analysis for marital¶
- There are more married customers followed by singles
- There are no null values
Education¶
In [1746]:
plt.figure(figsize=(5,5))
sns.countplot(df['education'],data=df)
plt.xlabel('Education', fontsize=15)
plt.ylabel('Count', fontsize=15)
plt.show();
In [1747]:
df['education'].unique() # Check unique values
df['education'].isnull().any() # Check null
df['education'].value_counts() # value counts
df['education'].value_counts(normalize=True) # value counts %age
Out[1747]:
Out[1747]:
Out[1747]:
Out[1747]:
Analysis for education¶
- There are maximum customers with secondary education followed by tertiary
- There are 1857 customers for which education details are not available
- Threre are no null values
Credit in default (default)¶
In [1748]:
plt.figure(figsize=(5,5))
sns.countplot(df['default'],data=df)
plt.xlabel('Credit in Default?', fontsize=15)
plt.ylabel('Count', fontsize=15)
plt.show();
In [1749]:
df['default'].unique() # Check unique values
df['default'].isnull().any() # Check null
df['default'].value_counts() # value counts
df['default'].value_counts(normalize=True) # value counts %age
Out[1749]:
Out[1749]:
Out[1749]:
Out[1749]:
Analysis for credit in default¶
- There are only 815 customers with credit in default
- Approximately 98% customers do not have credit in default
- There are no null values
Housing Loan (housing)¶
In [1750]:
plt.figure(figsize=(5,5))
sns.countplot(df['housing'],data=df)
plt.xlabel('Housing Loan?', fontsize=15)
plt.ylabel('Count', fontsize=15)
plt.show();
In [1751]:
df['housing'].unique() # Check unique values
df['housing'].isnull().any() # Check null
df['housing'].value_counts() # value counts
df['housing'].value_counts(normalize=True) # value counts %age
Out[1751]:
Out[1751]:
Out[1751]:
Out[1751]:
Analysis for housing loan¶
- There are more customers who have housing loan
- There are no null values
Personal Loan (loan)¶
In [1752]:
plt.figure(figsize=(5,5))
sns.countplot(df['loan'],data=df)
plt.xlabel('Personal Loan?', fontsize=15)
plt.ylabel('Count', fontsize=15)
plt.show();
In [1753]:
df['loan'].unique() # Check unique values
df['loan'].isnull().any() # Check null
df['loan'].value_counts() # value counts
df['loan'].value_counts(normalize=True) # value counts %age
Out[1753]:
Out[1753]:
Out[1753]:
Out[1753]:
Analysis for personal loan¶
- There are more customers (~84%) who do not have personal loan
- There are no null values
Balance in account (balance)¶
In [1754]:
plt.figure(figsize=(14,5))
sns.distplot(df['balance']);
In [1755]:
df['balance'].isnull().any() # Check null
df['balance'].describe() # Check mean, min, max, std, quartiles
df['balance'].median() # Median value
Out[1755]:
Out[1755]:
Out[1755]:
- Mean for balance is 1362.272058
- Median or Q2 for balance is 448.9
- Min value for balance is -8019
- Max value for balance is 102127
- Q1 for balance is 72
- Q2 for balance is 448
- Q3 for balance is 1428
- We see in plot above data is right skewed
In [1756]:
# Q1 and Q3 values from above
outliers_lower=72-1.5*(1428-72)
outliers_upper=1428+1.5*(1428-72)
print(outliers_lower)
print(outliers_upper)
# Number of outliers
print('Number of outliers lower:',df[df['balance']<outliers_lower]['balance'].count()) # Lower
print('Number of outliers upper:',df[df['balance']>outliers_upper]['balance'].count()) # Upper
In [1757]:
plt.figure(figsize=(7,7))
sns.boxplot(x='balance',data=df, orient='v');
Analysis for balance¶
- The minimum balance is -8019 and maximum balance is 102127
- There is much variance between mean and median values and we do not see any null values
- The data for balance is highly right skewed as can be seen from distplot and boxplot above
- As we can see from above the upper boundary for outliers is 3462 and there are 4712 outliers in upper boundary
Contact type (contact)¶
In [1758]:
plt.figure(figsize=(5,5))
sns.countplot(df['contact'],data=df)
plt.xlabel('Contact Type', fontsize=15)
plt.ylabel('Count', fontsize=15)
plt.show();
In [1759]:
df['contact'].unique() # Check unique values
df['contact'].isnull().any() # Check null
df['contact'].value_counts() # value counts
df['contact'].value_counts(normalize=True) # value counts %age
Out[1759]:
Out[1759]:
Out[1759]:
Out[1759]:
Analysis for contact type¶
- There are more customers (~65%) who were contacted on mobile
- However for 13020 customers (~6%) contact type is unknown
- There are no null values
Last contact (day, month)¶
In [1760]:
df['day'].unique() # Check unique values
df['day'].isnull().any() # Check null
df['day'].value_counts() # value counts
df['day'].value_counts(normalize=True) # value counts %age
Out[1760]:
Out[1760]:
Out[1760]:
Out[1760]:
In [1761]:
df['month'].unique() # Check unique values
df['month'].isnull().any() # Check null
df['month'].value_counts() # value counts
df['month'].value_counts(normalize=True) # value counts %age
Out[1761]:
Out[1761]:
Out[1761]:
Out[1761]:
Analysis for contact month¶
- Maximum contacts were made in the month of May and maximum contacts range from May to August
- There are no null values
Number of contacts (campaign)¶
In [1762]:
plt.figure(figsize=(14,5))
sns.distplot(df['campaign']);
In [1763]:
df['campaign'].isnull().any() # Check null
df['campaign'].describe() # Check mean, min, max, std, quartiles
df['campaign'].median() # Median value
Out[1763]:
Out[1763]:
Out[1763]:
- Mean for campaign is 2.763841
- Median or Q2 for campaign is 2
- Min value for campaign is 1
- Max value for campaign is 63
- Q1 for campaign is 1
- Q2 for campaign is 2
- Q3 for campaign is 3
- We see in the plot above data is right skewed
In [1764]:
# Q1 and Q3 values from above
outliers_lower=1-1.5*(3-1)
outliers_upper=3+1.5*(3-1)
print(outliers_lower)
print(outliers_upper)
# Number of outliers
print('Number of outliers lower:',df[df['balance']<outliers_lower]['balance'].count()) # Lower
print('Number of outliers upper:',df[df['balance']>outliers_upper]['balance'].count()) # Upper
Analysis for number of contacts (campaign)¶
- There is not much difference between mean and median values
- Minimum and Maximum for campaign are 1 and 63
- Data is highly right skewed
Days since last contact (pdays)¶
In [1765]:
df['pdays'].isnull().any() # Check null
df['pdays'].describe() # Check mean, min, max, std, quartiles
df['pdays'].median() # Median value
Out[1765]:
Out[1765]:
Out[1765]:
In [1766]:
print('pdays is -1: {}'.format(df[df.pdays==-1].shape[0]))
We see that for majority of records the pdays value is -1 which means either the customer has never been contacted or it is more than 900 since last contact
Number of contacts performed¶
In [1767]:
df['previous'].isnull().any() # Check null
df['previous'].describe() # Check mean, min, max, std, quartiles
df['previous'].median() # Median value
Out[1767]:
Out[1767]:
Out[1767]:
Outcome of previous campaign¶
In [1768]:
plt.figure(figsize=(5,5))
sns.countplot(df['poutcome'],data=df)
plt.xlabel('Previous Campaign Outcome', fontsize=15)
plt.ylabel('Count', fontsize=15)
plt.show();
In [1769]:
df['poutcome'].unique() # Check unique values
df['poutcome'].isnull().any() # Check null
df['poutcome'].value_counts() # value counts
df['poutcome'].value_counts(normalize=True) # value counts %age
Out[1769]:
Out[1769]:
Out[1769]:
Out[1769]:
As we see from above the success rate from previous campaign was around 3% though for a majority of customers it is not known (81.74%)
Did client subscribed (Target)¶
In [1770]:
plt.figure(figsize=(5,5))
sns.countplot(df['Target'],data=df)
plt.xlabel('Did Client Subscribed Term Deposit', fontsize=15)
plt.ylabel('Count', fontsize=15)
plt.show();
In [1771]:
df['Target'].unique() # Check unique values
df['Target'].isnull().any() # Check null
df['Target'].value_counts() # value counts
df['Target'].value_counts(normalize=True) # value counts %age
Out[1771]:
Out[1771]:
Out[1771]:
Out[1771]:
As we see from above around 12% customers subscribed to term deposits and there is some imbalance in out target variable¶
Address different data challenges¶
Address unknown values¶
In [1772]:
# So far we have seen unknown/mising data for job and education which can impact our target variable
# Let's check job data again
df['job'].value_counts()
Out[1772]:
In [1773]:
# From above we see for 288 customers we do not have their job information
# Let's check the number of customers who are above 60
df['job'][df['age']>60][df['job']=='unknown'].value_counts()
Out[1773]:
In [1774]:
# As we see from above 22 customers have no job information and are above the age of 60
# hence, we can move them to retired category
df.loc[(df['age']>60) & (df['job']=='unknown'), 'job'] = 'retired'
In [1775]:
# Let's compare job with education to infer if there a corerelation and we can identify/infer unknown values
pd.crosstab(df['job'], df['education'])
Out[1775]:
In [1776]:
# As we can infer from above, in case a person has primary education only then mostly they have blue-collar jobs
# Similarly, we see a person with tertiary education mostly have management jobs
# We are unable to infer similar undertstanding for people with secondary education as the spread is across technician,
# blue-collar and admin. jobs
# Hence, moving customers with primary education and unknown job to blue-collar job
# and customers with tertiary education and unknown job to management job
df.loc[(df['education']=='primary') & (df['job']=='unknown'), 'job'] = 'blue-collar'
df.loc[(df['education']=='tertiary') & (df['job']=='unknown'), 'job'] = 'management'
In [1777]:
pd.crosstab(df['education'], df['job'])
Out[1777]:
In [1778]:
# As we can see above, we can infer below understanding
# --> People in admin. job mostly have secondary education
# --> People in management job mostly have tertiary education
# --> People in services job mostly have secondary education
# --> People in technician job mostly have secondary education
# --> People in housemaid job mostly have primary education
# Hence, making changes as per this understanding/inference
df.loc[(df['job']=='admin.') & (df['education']=='unknown'), 'education'] = 'secondary'
df.loc[(df['job']=='management') & (df['education']=='unknown'), 'education'] = 'tertiary'
df.loc[(df['job']=='services') & (df['education']=='unknown'), 'education'] = 'secondary'
df.loc[(df['job']=='technician') & (df['education']=='unknown'), 'education'] = 'secondary'
df.loc[(df['job']=='housemaid') & (df['education']=='unknown'), 'education'] = 'primary'
Club into fewer classes¶
In [1779]:
# Categorize jobs and club into fewer classes
# We can club admin into management, housemaid into blue-collar and self employed with entrepreneur
mapping = {'admin.':'management', 'housemaid':'blue-collar', 'self-employed':'entrepreneur'}
df['job'] = df['job'].replace(mapping)
df['job'].value_counts()
df['job'].value_counts(normalize=True) # value counts %age
df['job']=df['job'].astype('category')
Out[1779]:
Out[1779]:
Address pdays values as -1¶
In [1780]:
# Earlier we saw that pdays had -1 for a huge number of records which means either the customer has never been contacted
# or it is more than 900 days
pd.crosstab(df['pdays'],df['poutcome'])
Out[1780]:
As we can see from above, pdays is -1 when poutcome is 'unknown'. It effectively means that pdays has -1 because customer has never been contacted. Let's break this up and categorize under different categories¶
In [1781]:
df['pdays_zero'] = 0
df['pdays_less_100'] = 0
df['pdays_great_100'] = 0
df['pdays_zero'][df['pdays']==-1] = 1
df['pdays_less_100'][(df['pdays']>-1) & (df['pdays']<=100)] = 1
df['pdays_great_100'][df['pdays']>100] = 1
df = df.drop('pdays', axis=1)
df.head()
Out[1781]:
Outliers¶
From our analysis above we saw outliers for age, balance and campaign. For age and campaign though there were outliers but there was not much variance between mean and median values and they seem to be real world situations, hence we will not treat them.
However, for balance we have a big difference in mean and median values, hence we will try to treat outliers for balance.
Outlier treatment for balance¶
From our analysis above, we had:
- The minimum balance is -8019 and maximum balance is 102127
- There is much variance between mean and median values and we do not see any null values
- The data for balance is highly right skewed
- Number of outliers upper: 4712
- Upper boundary for outliers: 3462
As we see above the number of upper range outliers is more than 10% as compared to overall data available, hence we need to transform for a better prediction model
In [1782]:
# Lets look at balance data again
plt.figure(figsize=(14,5))
sns.distplot(df['balance']);
As we see -ve balances in the dataset, we can safely mark them as zero balance as customers with higher balance only will opt for term deposits. Hence, converting -ve balances to zero. Also, balance data is right skewed, and we are taking a hypothesis that customers with higher balances will opt for term deposits. We are converting all balances more than upper outlier boundary to upper boundary.
In [1783]:
df.loc[df.balance<0,'balance'] = 0
In [1784]:
df.loc[df.balance>3462,'balance'] = 3462
Multivariate analysis¶
In [1785]:
plt.figure(figsize=(10,8))
sns.heatmap(df.corr(),
annot=True,
linewidths=.5,
center=0,
cbar=False,
cmap="YlGnBu")
plt.show();
In [1786]:
plt.figure(figsize=(20,5))
sns.pairplot(df, diag_kind='kde', diag_kws={'bw':'1.0'})
plt.show();
Target vs Age¶
In [1787]:
# Target vs Age
sns.boxplot(x='Target', y='age', data=df)
plt.show();
In [1788]:
# Target vs Age
sns.barplot(x='Target', y='age', data=df)
plt.show();
- Target variable does seem to be that much impacted by age
Target vs balance¶
In [1789]:
# Target vs balance
sns.boxplot(x='Target', y='balance', data=df)
plt.show();
In [1790]:
# Target vs balance
sns.barplot(x='Target', y='balance', data=df)
plt.show();
- We do see that customers with higher balance have more term deposits
Target vs duration¶
In [1791]:
sns.barplot(x='Target', y='duration', data=df)
plt.show();
In [1792]:
sns.boxplot(x='Target', y='duration', data=df)
plt.show();
In [1793]:
# As we can see from the plot above where duration of the call is longer more customers subscribed
# which is understandable as the subscription process would have taken more time
# it doesn't really reflect whether a longer duration impacted the decision of customer
# it seems the other way where a customer made a positive decision and hence a longer call duration
# dropping 'duration' as it's value will be known after call and doesn't seem to be impacting target variable for prediction
df=df.drop(['duration'],axis=1)
Target vs job¶
In [1794]:
# Target vs Job
pd.crosstab(df['job'], df['Target'], normalize='index')
Out[1794]:
In [1795]:
plt.figure(figsize=(20,5))
sns.countplot(x='job',hue='Target', data=df);
As we see from above highest percentages for customers accepting term deposit are (converion rate):
- Student subscribed to term deposit = 28.67%
- Retired subscribed to term deposit = 22.79%
- Unempld subscribed to term deposit = 15.50%
Overall people with job as management have more term deposits followed by technician and blue-collar jobs
Target vs marital¶
In [1796]:
# Target vs Marital
pd.crosstab(df['marital'], df['Target'], normalize='index')
Out[1796]:
In [1797]:
sns.countplot(x='marital',hue='Target', data=df);
As we see from above highest percentages for customers accepting term deposit are (conversion rate):
- Single subscribed to term deposit = 14.94%
- Divorced subscribed to term deposit = 11.94%
Overall married customers have more term deposits followed by single customers
Target vs Education¶
In [1798]:
# Target vs Education
pd.crosstab(df['education'], df['Target'], normalize='index')
Out[1798]:
In [1799]:
sns.countplot(x='education',hue='Target', data=df);
As we see from above highest percentages for customers accepting term deposit are (conversion rate):
- Customers with Tertiary(higher) education subscribed to term deposit = 15%
- Next highest %age is for customers for whom we do not know their education = 13.57%
Overall customers with secondary education have more term deposits followed by tertiary education
Target vs credit default¶
In [1800]:
# Target vs Default
pd.crosstab(df['default'], df['Target'], normalize='index')
Out[1800]:
In [1801]:
sns.countplot(x='default',hue='Target', data=df);
As we see from above customers who do not default on credit have a higher %age of accepting term deposit
Target vs Housing loan¶
In [1802]:
# Target vs Housing
pd.crosstab(df['housing'], df['Target'], normalize='index')
Out[1802]:
In [1803]:
sns.countplot(x='housing',hue='Target', data=df);
As we see from above customers who do not have home loan had higher %age accepting term deposits
Target vs Personal Loan¶
In [1804]:
# Target vs Loan
pd.crosstab(df['loan'], df['Target'], normalize='index')
Out[1804]:
In [1805]:
sns.countplot(x='loan',hue='Target', data=df);
As we see from above customers who do not have personal loan had higher %age accepting term deposit
Target vs Contact¶
In [1806]:
# Target vs Contact
pd.crosstab(df['contact'], df['Target'])
Out[1806]:
In [1807]:
sns.countplot(x='contact',hue='Target', data=df);
We see from above that there are 13020 records where contact is Unkown. From that count 530 have Target variable as 'yes'. We also see that for majority where the Target variable is 'yes', the contact type was cellular. Hence, we can safely move unknown contacts where Target is 'yes' to contact as 'cellular'
In [1808]:
df.loc[(df['contact']=='unknown') & (df['Target']=='yes'), 'contact'] = 'cellular'
Encode categorical variables¶
In [1809]:
le = preprocessing.LabelEncoder()
df.job = le.fit_transform(df.job)
df.marital = le.fit_transform(df.marital)
df.education = le.fit_transform(df.education)
df.default = le.fit_transform(df.default)
df.housing = le.fit_transform(df.housing)
df.loan = le.fit_transform(df.loan)
df.contact = le.fit_transform(df.contact)
df.month = le.fit_transform(df.month)
df.poutcome = le.fit_transform(df.poutcome)
df.Target = le.fit_transform(df.Target)
df.describe().T
Out[1809]:
Let's handle class imbalance in target variable by over sampling¶
In [1810]:
df['Target'].value_counts(normalize=True)
df_copy=df.copy()
df_2copy=df_copy[df.Target==1]
df_copy=pd.concat([df_copy, df_2copy])
df_copy=pd.concat([df_copy, df_2copy])
df_copy=pd.concat([df_copy, df_2copy])
df_copy=pd.concat([df_copy, df_2copy])
df_copy=pd.concat([df_copy, df_2copy])
df_copy=pd.concat([df_copy, df_2copy])
df_copy=pd.concat([df_copy, df_2copy])
df=df_copy
df['Target'].value_counts(normalize=True)
Out[1810]:
Out[1810]:
Split data and create Models¶
In [1811]:
x=df.drop('Target',axis=1) # Independent variables
y=df['Target'] # Dependent variable
features = [col for col in df.columns if col != 'Target']
In [1812]:
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.30,random_state=1) # Split data in 70:30 ratio
Normalize/Scale¶
In [1813]:
from sklearn.preprocessing import StandardScaler
scaler=StandardScaler()
scaler.fit(x_train)
x_train = scaler.transform(x_train)
x_test = scaler.transform(x_test)
Out[1813]:
Logistic Regression¶
In [1814]:
logreg = LogisticRegression(random_state=1)
logreg.fit(x_train, y_train) # Fit training dataset
Out[1814]:
In [1878]:
print("Train: %.2f" % logreg.score(x_train, y_train)) # performance on train data
print("Test: %.2f" % logreg.score(x_test, y_test)) # performance on test data
In [1816]:
y_predict = logreg.predict(x_test) # Get predicted
In [1817]:
# Confusion matrix
pd.crosstab(y_test, y_predict, rownames=['Actual'], colnames=['Predicted'])
Out[1817]:
In [1818]:
# Check different metrics
print('Confusion Matrix')
print(confusion_matrix(y_test,y_predict))
print()
var_recall=recall_score(y_test,y_predict)
print("Recall:",var_recall)
var_precision=precision_score(y_test,y_predict)
print("Precision:",var_precision)
var_f1=f1_score(y_test,y_predict)
print("F1 Score:",var_f1)
var_roc=roc_auc_score(y_test,y_predict)
print("Roc Auc Score:",var_roc)
var_accuracy=accuracy_score(y_test,y_predict)
print("Accuracy Score:",var_accuracy)
print()
print(classification_report(y_test, y_predict))
Perform Grid search to get best params for Logistic Regression Model¶
In [1819]:
from sklearn.model_selection import RandomizedSearchCV
penalty = ['l1', 'l2']
C = [0.01,0.1,0.25,0.5,0.75,1]
solver = ['newton-cg','lbfgs','liblinear','sag','saga']
hyperparameters = dict(C=C, penalty=penalty, solver = solver)
logistic = LogisticRegression()
final_model = RandomizedSearchCV(logistic, hyperparameters, verbose=0)
final_model.fit(x_train,y_train)
y_predict = final_model.predict(x_test)
Out[1819]:
In [1820]:
# Check different metrics
print('Confusion Matrix')
print(confusion_matrix(y_test,y_predict))
print()
var_recall=recall_score(y_test,y_predict)
print("Recall:",var_recall)
var_precision=precision_score(y_test,y_predict)
print("Precision:",var_precision)
var_f1=f1_score(y_test,y_predict)
print("F1 Score:",var_f1)
var_roc=roc_auc_score(y_test,y_predict)
print("Roc Auc Score:",var_roc)
var_accuracy=accuracy_score(y_test,y_predict)
print("Accuracy Score:",var_accuracy)
print()
print(classification_report(y_test, y_predict))
Put results in a dataframe for comparison later¶
In [1821]:
#Store the accuracy results for each model in a dataframe for final comparison
resultsDf = pd.DataFrame({'Method':['Logistic Regression'], 'Accuracy': var_accuracy, 'Recall': var_recall,'Precision': var_precision,'F1 Score': var_f1,'ROC AUC Score': var_roc})
resultsDf = resultsDf[['Method', 'Accuracy','Recall','Precision','F1 Score','ROC AUC Score']]
resultsDf
Out[1821]:
Decision Tree¶
In [1822]:
dt = DecisionTreeClassifier(criterion = 'entropy',random_state=1 )
dt.fit(x_train, y_train)
Out[1822]:
In [1877]:
print("Train: %.2f" % dt.score(x_train, y_train)) # performance on train data
print("Test: %.2f" % dt.score(x_test, y_test)) # performance on test data
There is a high degree of overfitting in the model due to which the test accuracy drops.¶
In [1824]:
dot_data = StringIO()
export_graphviz(dt, out_file=dot_data,
filled=True, rounded=True,
special_characters=True,feature_names = features,class_names=['0','1'])
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
graph.write_png('TD_Sales.png')
Image(graph.create_png())
Out[1824]:
Out[1824]:
In [1825]:
y_predict = dt.predict(x_test)
In [1826]:
# Confusion matrix
pd.crosstab(y_test, y_predict, rownames=['Actual'], colnames=['Predicted'])
Out[1826]:
In [1827]:
# Check different metrics
print('Confusion Matrix')
print(confusion_matrix(y_test,y_predict))
var_recall=recall_score(y_test,y_predict)
print("Recall:",var_recall)
var_precision=precision_score(y_test,y_predict)
print("Precision:",var_precision)
var_f1=f1_score(y_test,y_predict)
print("F1 Score:",var_f1)
var_roc=roc_auc_score(y_test,y_predict)
print("Roc Auc Score:",var_roc)
var_accuracy=accuracy_score(y_test,y_predict)
print("Accuracy Score:",var_accuracy)
print()
print(classification_report(y_test, y_predict))
In [1828]:
tempResultsDf = pd.DataFrame({'Method':['Decision Tree'], 'Accuracy': var_accuracy, 'Recall': var_recall,'Precision': var_precision,'F1 Score': var_f1,'ROC AUC Score': var_roc})
resultsDf = pd.concat([resultsDf, tempResultsDf])
resultsDf = resultsDf[['Method', 'Accuracy','Recall','Precision','F1 Score','ROC AUC Score']]
resultsDf
Out[1828]:
Prune the Decision Tree¶
In [1915]:
dt_pruned = DecisionTreeClassifier(criterion = "entropy", max_depth=6)
dt_pruned.fit(x_train, y_train)
Out[1915]:
In [1916]:
print("Train: %.2f" % dt_pruned.score(x_train, y_train)) # performance on train data
print("Test: %.2f" % dt_pruned.score(x_test, y_test)) # performance on test data
When the tree is regularized, overfitting is reduced.¶
In [1831]:
dot_data = StringIO()
export_graphviz(dt_pruned, out_file=dot_data,
filled=True, rounded=True,
special_characters=True,feature_names = features,class_names=['0','1'])
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
graph.write_png('TD_Sales_Pruned.png')
Image(graph.create_png())
Out[1831]:
Out[1831]:
In [1832]:
y_predict = dt_pruned.predict(x_test)
In [1833]:
# Confusion matrix
pd.crosstab(y_test, y_predict, rownames=['Actual'], colnames=['Predicted'])
Out[1833]:
In [1834]:
# Check different metrics
print('Confusion Matrix')
print(confusion_matrix(y_test,y_predict))
var_recall=recall_score(y_test,y_predict)
print("Recall:",var_recall)
var_precision=precision_score(y_test,y_predict)
print("Precision:",var_precision)
var_f1=f1_score(y_test,y_predict)
print("F1 Score:",var_f1)
var_roc=roc_auc_score(y_test,y_predict)
print("Roc Auc Score:",var_roc)
var_accuracy=accuracy_score(y_test,y_predict)
print("Accuracy Score:",var_accuracy)
print()
print(classification_report(y_test, y_predict))
Calculating the feature importance¶
In [1835]:
feat_importance = dt_pruned.tree_.compute_feature_importances(normalize=False)
feat_imp_dict = dict(zip(features, dt_pruned.feature_importances_))
feat_imp = pd.DataFrame.from_dict(feat_imp_dict, orient='index')
feat_imp.sort_values(by=0, ascending=False)
Out[1835]:
From above table we can infer that contact and month impact our target variable most.
Insert into result data frame¶
In [1836]:
tempResultsDf = pd.DataFrame({'Method':['Pruned Decision Tree'], 'Accuracy': var_accuracy, 'Recall': var_recall,'Precision': var_precision,'F1 Score': var_f1,'ROC AUC Score': var_roc})
resultsDf = pd.concat([resultsDf, tempResultsDf])
resultsDf = resultsDf[['Method', 'Accuracy','Recall','Precision','F1 Score','ROC AUC Score']]
resultsDf
Out[1836]:
Random Forest¶
In [1837]:
rfcl = RandomForestClassifier(n_estimators = 50)
rfcl = rfcl.fit(x_train, y_train)
In [1838]:
print("Train: %.2f" % rfcl.score(x_train, y_train)) # performance on train data
print("Test: %.2f" % rfcl.score(x_test, y_test)) # performance on test data
In [1839]:
y_predict = rfcl.predict(x_test)
In [1840]:
# Confusion matrix
pd.crosstab(y_test, y_predict, rownames=['Actual'], colnames=['Predicted'])
Out[1840]:
In [1841]:
# Check different metrics
print('Confusion Matrix')
print(confusion_matrix(y_test,y_predict))
var_recall=recall_score(y_test,y_predict)
print("Recall:",var_recall)
var_precision=precision_score(y_test,y_predict)
print("Precision:",var_precision)
var_f1=f1_score(y_test,y_predict)
print("F1 Score:",var_f1)
var_roc=roc_auc_score(y_test,y_predict)
print("Roc Auc Score:",var_roc)
var_accuracy=accuracy_score(y_test,y_predict)
print("Accuracy Score:",var_accuracy)
print()
print(classification_report(y_test, y_predict))
In [1842]:
n_estimators = [int(x) for x in np.linspace(start = 200, stop = 2000, num = 10)]
# Number of features to consider at every split
max_features = ['auto', 'sqrt']
# Maximum number of levels in tree
max_depth = [int(x) for x in np.linspace(10, 110, num = 11)]
max_depth.append(None)
# Minimum number of samples required to split a node
min_samples_split = [2, 5, 10]
# Minimum number of samples required at each leaf node
min_samples_leaf = [1, 2, 4]
# Method of selecting samples for training each tree
bootstrap = [True, False]# Create the random grid
random_grid = {'n_estimators': n_estimators,
'max_features': max_features,
'max_depth': max_depth,
'min_samples_split': min_samples_split,
'min_samples_leaf': min_samples_leaf,
'bootstrap': bootstrap}
print(random_grid)
Perform random search to get best model¶
In [1843]:
rf = RandomForestClassifier()
# Random search of parameters, using 3 fold cross validation,
# search across 100 different combinations, and use all available cores
rf_random = RandomizedSearchCV(estimator = rf, param_distributions = random_grid, n_iter = 100, cv = 3, verbose=2, random_state=42, n_jobs = -1)# Fit the random search model
rf_random.fit(x_train, y_train)
Out[1843]:
Check best parameters¶
In [1844]:
rf_random.best_params_
Out[1844]:
Predict using final model¶
In [1845]:
y_predict = rf_random.predict(x_test)
In [1846]:
# Confusion matrix
pd.crosstab(y_test, y_predict, rownames=['Actual'], colnames=['Predicted'])
Out[1846]:
In [1847]:
# Check different metrics
print('Confusion Matrix')
print(confusion_matrix(y_test,y_predict))
var_recall=recall_score(y_test,y_predict)
print("Recall:",var_recall)
var_precision=precision_score(y_test,y_predict)
print("Precision:",var_precision)
var_f1=f1_score(y_test,y_predict)
print("F1 Score:",var_f1)
var_roc=roc_auc_score(y_test,y_predict)
print("Roc Auc Score:",var_roc)
var_accuracy=accuracy_score(y_test,y_predict)
print("Accuracy Score:",var_accuracy)
print()
print(classification_report(y_test, y_predict))
Insert into result data frame¶
In [1848]:
tempResultsDf = pd.DataFrame({'Method':['Random Forest'], 'Accuracy': var_accuracy, 'Recall': var_recall,'Precision': var_precision,'F1 Score': var_f1,'ROC AUC Score': var_roc})
resultsDf = pd.concat([resultsDf, tempResultsDf])
resultsDf = resultsDf[['Method', 'Accuracy','Recall','Precision','F1 Score','ROC AUC Score']]
resultsDf
Out[1848]:
Bagging¶
In [1849]:
bgcl = BaggingClassifier(n_estimators=50, max_samples= .5, bootstrap=True, oob_score=True, random_state=22)
bgcl = bgcl.fit(x_train, y_train)
In [1850]:
print("Train: %.2f" % bgcl.score(x_train, y_train)) # performance on train data
print("Test: %.2f" % bgcl.score(x_test, y_test)) # performance on test data
In [1851]:
y_predict = bgcl.predict(x_test)
In [1852]:
# Confusion matrix
pd.crosstab(y_test, y_predict, rownames=['Actual'], colnames=['Predicted'])
Out[1852]:
In [1853]:
# Check different metrics
print('Confusion Matrix')
print(confusion_matrix(y_test,y_predict))
var_recall=recall_score(y_test,y_predict)
print("Recall:",var_recall)
var_precision=precision_score(y_test,y_predict)
print("Precision:",var_precision)
var_f1=f1_score(y_test,y_predict)
print("F1 Score:",var_f1)
var_roc=roc_auc_score(y_test,y_predict)
print("Roc Auc Score:",var_roc)
var_accuracy=accuracy_score(y_test,y_predict)
print("Accuracy Score:",var_accuracy)
print()
print(classification_report(y_test, y_predict))
In [1854]:
n_estimators = [int(x) for x in np.linspace(start = 200, stop = 2000, num = 10)]
# Method of selecting samples for training each tree
bootstrap = [True, False]# Create the random grid
random_grid = {'n_estimators': n_estimators,
'bootstrap': bootstrap}
print(random_grid)
Perform random search for best model¶
In [1855]:
bgcl = BaggingClassifier()
# Random search of parameters, using 3 fold cross validation,
# search across 100 different combinations, and use all available cores
bgcl_random = RandomizedSearchCV(estimator = bgcl, param_distributions = random_grid, n_iter = 100, cv = 3, verbose=2, random_state=42, n_jobs = -1)# Fit the random search model
bgcl_random.fit(x_train, y_train)
Out[1855]:
Check best parameters¶
In [1856]:
bgcl_random.best_params_
Out[1856]:
Predict using final model¶
In [1857]:
y_predict = bgcl_random.predict(x_test)
In [1858]:
# Check different metrics
print('Confusion Matrix')
print(confusion_matrix(y_test,y_predict))
var_recall=recall_score(y_test,y_predict)
print("Recall:",var_recall)
var_precision=precision_score(y_test,y_predict)
print("Precision:",var_precision)
var_f1=f1_score(y_test,y_predict)
print("F1 Score:",var_f1)
var_roc=roc_auc_score(y_test,y_predict)
print("Roc Auc Score:",var_roc)
var_accuracy=accuracy_score(y_test,y_predict)
print("Accuracy Score:",var_accuracy)
print()
print(classification_report(y_test, y_predict))
Update results data frame¶
In [1859]:
tempResultsDf = pd.DataFrame({'Method':['Bagging'], 'Accuracy': var_accuracy, 'Recall': var_recall,'Precision': var_precision,'F1 Score': var_f1,'ROC AUC Score': var_roc})
resultsDf = pd.concat([resultsDf, tempResultsDf])
resultsDf = resultsDf[['Method', 'Accuracy','Recall','Precision','F1 Score','ROC AUC Score']]
resultsDf
Out[1859]:
Adaboosting¶
In [1860]:
abcl = AdaBoostClassifier(n_estimators = 100, learning_rate=0.1, random_state=22)
abcl = abcl.fit(x_train, y_train)
In [1861]:
print("Train: %.2f" % abcl.score(x_train, y_train)) # performance on train data
print("Test: %.2f" % abcl.score(x_test, y_test)) # performance on test data
In [1862]:
y_predict = abcl.predict(x_test)
In [1863]:
# Confusion matrix
pd.crosstab(y_test, y_predict, rownames=['Actual'], colnames=['Predicted'])
Out[1863]:
In [1864]:
# Check different metrics
print('Confusion Matrix')
print(confusion_matrix(y_test,y_predict))
var_recall=recall_score(y_test,y_predict)
print("Recall:",var_recall)
var_precision=precision_score(y_test,y_predict)
print("Precision:",var_precision)
var_f1=f1_score(y_test,y_predict)
print("F1 Score:",var_f1)
var_roc=roc_auc_score(y_test,y_predict)
print("Roc Auc Score:",var_roc)
var_accuracy=accuracy_score(y_test,y_predict)
print("Accuracy Score:",var_accuracy)
print()
print(classification_report(y_test, y_predict))
In [1865]:
n_estimators = [int(x) for x in np.linspace(start = 200, stop = 2000, num = 10)]
# Method of selecting samples for training each tree
random_grid = {'n_estimators': n_estimators}
print(random_grid)
Random search to get best model¶
In [1866]:
abcl = BaggingClassifier()
# Random search of parameters, using 3 fold cross validation,
# search across 100 different combinations, and use all available cores
abcl_random = RandomizedSearchCV(estimator = abcl, param_distributions = random_grid, n_iter = 100, cv = 3, verbose=2, random_state=42, n_jobs = -1)# Fit the random search model
abcl_random.fit(x_train, y_train)
Out[1866]:
Predict using model as per ramdomized search¶
In [1868]:
y_predict = abcl_random.predict(x_test)
In [1869]:
# Check different metrics
print('Confusion Matrix')
print(confusion_matrix(y_test,y_predict))
var_recall=recall_score(y_test,y_predict)
print("Recall:",var_recall)
var_precision=precision_score(y_test,y_predict)
print("Precision:",var_precision)
var_f1=f1_score(y_test,y_predict)
print("F1 Score:",var_f1)
var_roc=roc_auc_score(y_test,y_predict)
print("Roc Auc Score:",var_roc)
var_accuracy=accuracy_score(y_test,y_predict)
print("Accuracy Score:",var_accuracy)
print()
print(classification_report(y_test, y_predict))
Update results data frame¶
In [1870]:
tempResultsDf = pd.DataFrame({'Method':['Adaboost'], 'Accuracy': var_accuracy, 'Recall': var_recall,'Precision': var_precision,'F1 Score': var_f1,'ROC AUC Score': var_roc})
resultsDf = pd.concat([resultsDf, tempResultsDf])
resultsDf = resultsDf[['Method', 'Accuracy','Recall','Precision','F1 Score','ROC AUC Score']]
resultsDf
Out[1870]:
GradientBoost¶
In [1871]:
gbcl = GradientBoostingClassifier(n_estimators = 50, learning_rate = 0.1, random_state=22)
gbcl = gbcl.fit(x_train, y_train)
In [1872]:
print("Train: %.2f" % gbcl.score(x_train, y_train)) # performance on train data
print("Test: %.2f" % gbcl.score(x_test, y_test)) # performance on test data
In [1873]:
y_predict = gbcl.predict(x_test)
In [1874]:
# Confusion matrix
pd.crosstab(y_test, y_predict, rownames=['Actual'], colnames=['Predicted'])
Out[1874]:
In [1875]:
# Check different metrics
print('Confusion Matrix')
print(confusion_matrix(y_test,y_predict))
var_recall=recall_score(y_test,y_predict)
print("Recall:",var_recall)
var_precision=precision_score(y_test,y_predict)
print("Precision:",var_precision)
var_f1=f1_score(y_test,y_predict)
print("F1 Score:",var_f1)
var_roc=roc_auc_score(y_test,y_predict)
print("Roc Auc Score:",var_roc)
var_accuracy=accuracy_score(y_test,y_predict)
print("Accuracy Score:",var_accuracy)
print()
print(classification_report(y_test, y_predict))
Update result data frame¶
In [1876]:
tempResultsDf = pd.DataFrame({'Method':['Gradient boost'], 'Accuracy': var_accuracy, 'Recall': var_recall,'Precision': var_precision,'F1 Score': var_f1,'ROC AUC Score': var_roc})
resultsDf = pd.concat([resultsDf, tempResultsDf])
resultsDf = resultsDf[['Method', 'Accuracy','Recall','Precision','F1 Score','ROC AUC Score']]
resultsDf
Out[1876]:
From the table above we see that Random Forest gives the best results after handling target class imbalance and scaling the data with an accuracy of 98% and with a good recall, precision and F1 score. Even the ROC AUC score is 98%¶
The model will give 98% correct prediction for customers who will subscribe to term deposit.
- The other models that performed better were Bagging and Adaboost which also give comparable results.